{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# More efficient broadcast of arrays with memmap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Data movement is where IPython's naive model suffers the most.\n", "But knowing about your cluster lets you make smarter decisions about data movement than a simple `rc[:].push`.\n", "\n", "I ran this example with a cluster on a 64-core remote VM,\n", "so communication between the client and controller is over the public internet,\n", "while communication between the controller and engines is local.\n", "\n", "This is an example of 'broadcasting' a numpy array using memmapped files,\n", "to reduce the amount of expensive network traffic when several engines are on the same host." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import socket\n", "import os, sys, re\n", "\n", "import numpy as np\n", "\n", "import ipyparallel as ipp" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "rc = ipp.Client(profile=\"mpi\")\n", "eall = rc.broadcast_view(coalescing=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, create a map of engine id to hostname" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{0: 'ip-172-31-2-77',\n", " 1: 'ip-172-31-2-77',\n", " 2: 'ip-172-31-2-77',\n", " 3: 'ip-172-31-2-77',\n", " 4: 'ip-172-31-2-77',\n", " 5: 'ip-172-31-2-77',\n", " 6: 'ip-172-31-2-77',\n", " 7: 'ip-172-31-2-77',\n", " 8: 'ip-172-31-2-77',\n", " 9: 'ip-172-31-2-77',\n", " 10: 'ip-172-31-2-77',\n", " 11: 'ip-172-31-2-77',\n", " 12: 'ip-172-31-2-77',\n", " 13: 'ip-172-31-2-77',\n", " 14: 'ip-172-31-2-77',\n", " 15: 'ip-172-31-2-77',\n", " 16: 'ip-172-31-2-77',\n", " 17: 'ip-172-31-2-77',\n", " 18: 'ip-172-31-2-77',\n", " 19: 'ip-172-31-2-77',\n", " 20: 'ip-172-31-2-77',\n", " 21: 'ip-172-31-2-77',\n", " 22: 'ip-172-31-2-77',\n", " 23: 'ip-172-31-2-77',\n", " 24: 'ip-172-31-2-77',\n", " 25: 'ip-172-31-2-77',\n", " 26: 'ip-172-31-2-77',\n", " 27: 'ip-172-31-2-77',\n", " 28: 'ip-172-31-2-77',\n", " 29: 'ip-172-31-2-77',\n", " 30: 'ip-172-31-2-77',\n", " 31: 'ip-172-31-2-77',\n", " 32: 'ip-172-31-2-77',\n", " 33: 'ip-172-31-2-77',\n", " 34: 'ip-172-31-2-77',\n", " 35: 'ip-172-31-2-77',\n", " 36: 'ip-172-31-2-77',\n", " 37: 'ip-172-31-2-77',\n", " 38: 'ip-172-31-2-77',\n", " 39: 'ip-172-31-2-77',\n", " 40: 'ip-172-31-2-77',\n", " 41: 'ip-172-31-2-77',\n", " 42: 'ip-172-31-2-77',\n", " 43: 'ip-172-31-2-77',\n", " 44: 'ip-172-31-2-77',\n", " 45: 'ip-172-31-2-77',\n", " 46: 'ip-172-31-2-77',\n", " 47: 'ip-172-31-2-77',\n", " 48: 'ip-172-31-2-77',\n", " 49: 'ip-172-31-2-77',\n", " 50: 'ip-172-31-2-77',\n", " 51: 'ip-172-31-2-77',\n", " 52: 'ip-172-31-2-77',\n", " 53: 'ip-172-31-2-77',\n", " 54: 'ip-172-31-2-77',\n", " 55: 'ip-172-31-2-77',\n", " 56: 'ip-172-31-2-77',\n", " 57: 'ip-172-31-2-77',\n", " 58: 'ip-172-31-2-77',\n", " 59: 'ip-172-31-2-77',\n", " 60: 'ip-172-31-2-77',\n", " 61: 'ip-172-31-2-77',\n", " 62: 'ip-172-31-2-77',\n", " 63: 'ip-172-31-2-77'}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "engine_hosts = eall.apply_async(socket.gethostname).get_dict()\n", "engine_hosts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, reverse that to create a map of hostname to engine ids" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ip-172-31-2-77': [0,\n", " 1,\n", " 2,\n", " 3,\n", " 4,\n", " 5,\n", " 6,\n", " 7,\n", " 8,\n", " 9,\n", " 10,\n", " 11,\n", " 12,\n", " 13,\n", " 14,\n", " 15,\n", " 16,\n", " 17,\n", " 18,\n", " 19,\n", " 20,\n", " 21,\n", " 22,\n", " 23,\n", " 24,\n", " 25,\n", " 26,\n", " 27,\n", " 28,\n", " 29,\n", " 30,\n", " 31,\n", " 32,\n", " 33,\n", " 34,\n", " 35,\n", " 36,\n", " 37,\n", " 38,\n", " 39,\n", " 40,\n", " 41,\n", " 42,\n", " 43,\n", " 44,\n", " 45,\n", " 46,\n", " 47,\n", " 48,\n", " 49,\n", " 50,\n", " 51,\n", " 52,\n", " 53,\n", " 54,\n", " 55,\n", " 56,\n", " 57,\n", " 58,\n", " 59,\n", " 60,\n", " 61,\n", " 62,\n", " 63]}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "host_engines = {}\n", "\n", "for eid, host in engine_hosts.items():\n", " if host not in host_engines:\n", " host_engines[host] = []\n", " host_engines[host].append(eid)\n", "\n", "host_engines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can measure our baseline overhead: how long does it take to roundrip an empty task on all engines.\n", "We shouldn't expect anything to take less time than this." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 165 ms, sys: 40.9 ms, total: 206 ms\n", "Wall time: 1.06 s\n" ] } ], "source": [ "%time _ = eall.apply_sync(lambda : None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's look at how long it takes to send data in the simplest possible way" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "data = np.random.random((512, 512))" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "694099ab95124e83b4442ae57a8178e8", "version_major": 2, "version_minor": 0 }, "text/plain": [ "_push: 0%| | 0/64 [00:00